A numerical study on learning curves in stochastic multilayer feedforward networks.
نویسندگان
چکیده
The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t2 scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.
منابع مشابه
Learning Stochastic Feedforward Neural Networks
Multilayer perceptrons (MLPs) or neural networks are popular models used for nonlinear regression and classification tasks. As regressors, MLPs model the conditional distribution of the predictor variables Y given the input variables X . However, this predictive distribution is assumed to be unimodal (e.g. Gaussian). For tasks involving structured prediction, the conditional distribution should...
متن کاملPattern Discrimination Using Feedforward Networks: A Benchmark Study of Scaling Behavior
The discrimination powers of Multilayer perceptron (MLP) and Learning Vector Quantisation (LVQ) networks are compared for overlapping Gaussian distributions. It is shown, both analytically and with Monte Carlo studies, that the MLP network handles high dimensional problems in a more eecient way than LVQ. This is mainly due to the sigmoidal form of the MLP transfer function, but also to the the ...
متن کاملLearning Curves for Stochastic Gradient Descent in Linear Feedforward Networks
Gradient-following learning methods can encounter problems of implementation in many applications, and stochastic variants are sometimes used to overcome these difficulties. We analyze three online training methods used with a linear perceptron: direct gradient descent, node perturbation, and weight perturbation. Learning speed is defined as the rate of exponential decay in the learning curves....
متن کاملAverage-Case Learning Curves for Radial Basis Function Networks
The application of statistical physics to the study of the learning curves of feedforward connectionist networks has, to date, been concerned mostly with networks that do not include hidden layers. Recent work has extended the theory to networks such as committee machines and parity machines; however these are not networks that are often used in practice and an important direction for current a...
متن کاملUsing the Taylor expansion of multilayer feedforward neural networks
The Taylor series expansion of continuous functions has shown in many fields to be an extremely powerful tool to study the characteristics of such functions. This paper illustrates the power of the Taylor series expansion of multilayer feedforward neural networks. The paper shows how these expansions can be used to investigate positions of decision boundaries, to develop active learning strateg...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
- Neural computation
دوره 8 5 شماره
صفحات -
تاریخ انتشار 1996